White balance and fixed pattern noise frame calibration using distal cap

ABSTRACT

The disclosure extends to methods, systems, and computer program products for producing an image in light deficient environments and correction of white balance and/or fixed pattern noise at startup or at any other time during a procedure.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.15/351,222 filed Nov. 14, 2016, which is a continuation of U.S. patentapplication Ser. No. 14/214,791 filed Mar. 15, 2014 (now U.S. Pat. No.9,492,060) and which claims the benefit of U.S. Provisional ApplicationNo. 61/791,186, filed Mar. 15, 2013, which are incorporated herein byreference in their entirety, including but not limited to those portionsthat specifically appear hereinafter, the incorporation by referencebeing made with the following exception: In the event that any portionof any of the above-referenced applications is inconsistent with thisapplication, this application supersedes said above-referencedapplications.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable.

BACKGROUND

Advances in technology have provided advances in imaging capabilitiesfor medical use. One area that has enjoyed some of the most beneficialadvances is that of endoscopic surgical procedures because of theadvances in the components that make up an endoscope.

The disclosure relates generally to electromagnetic sensing and sensors,increasing the color accuracy and reducing the fixed pattern noise. Thefeatures and advantages of the disclosure will be set forth in thedescription which follows, and in part will be apparent from thedescription, or may be learned by the practice of the disclosure withoutundue experimentation. The features and advantages of the disclosure maybe realized and obtained by means of the instruments and combinationsparticularly pointed out in the appended claims.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive implementations of the disclosure aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified. Advantages of the disclosure will becomebetter understood with regard to the following description andaccompanying drawings.

FIG. 1 illustrates various embodiments of an endoscopic system inaccordance with the principles and teachings of the disclosure;

FIG. 2A is an illustration of a distal cap in accordance with theprinciples and teachings of the disclosure;

FIG. 2B is an illustration of a distal cap in accordance with theprinciples and teachings of the disclosure;

FIG. 3 illustrates a schematic of supporting and enabling hardware inaccordance with the principles and teachings of the disclosure;

FIGS. 4A and 4B illustrate a perspective view and a side view,respectively, of an implementation of a monolithic sensor having aplurality of pixel arrays for producing a three dimensional image inaccordance with the teachings and principles of the disclosure;

FIGS. 5A and 5B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor built on aplurality of substrates, wherein a plurality of pixel columns formingthe pixel array are located on the first substrate and a plurality ofcircuit columns are located on a second substrate and showing anelectrical connection and communication between one column of pixels toits associated or corresponding column of circuitry;

FIGS. 6A and 6B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor having aplurality of pixel arrays for producing a three dimensional image,wherein the plurality of pixel arrays and the image sensor are built ona plurality of substrates;

FIG. 7 is a flow chart illustrating an implementation of a system andmethod for adjusting white balance in accordance with the principles andteachings of the disclosure;

FIG. 8 is a flow chart illustrating an implementation of a system andmethod for adjusting white balance in accordance with the principles andteachings of the disclosure; and

FIG. 9 is a flow chart illustrating an implementation of a system andmethod for adjusting white balance in accordance with the principles andteachings of the disclosure.

DETAILED DESCRIPTION

The disclosure extends to methods, systems, and computer based productsfor digital imaging that may be primarily suited to medicalapplications, and for producing an image in light deficient environmentsand correction of white balance and/or fixed pattern noise at startup orat any other time during a procedure.

In the following description of the disclosure, reference is made to theaccompanying drawings, which form a part hereof, and in which is shownby way of illustration specific implementations in which the disclosuremay be practiced. It is understood that other implementations may beutilized and structural changes may be made without departing from thescope of the disclosure.

In describing and claiming the subject matter of the disclosure, thefollowing terminology will be used in accordance with the definitionsset out below.

It must be noted that, as used in this specification and the appendedclaims, the singular forms “a,” “an,” and “the” include plural referentsunless the context clearly dictates otherwise.

As used herein, the terms “comprising,” “including,” “containing,”“characterized by,” and grammatical equivalents thereof are inclusive oropen-ended terms that do not exclude additional, unrecited elements ormethod steps.

As used herein, the phrase “consisting of” and grammatical equivalentsthereof exclude any element or step not specified in the claim.

As used herein, the phrase “consisting essentially of” and grammaticalequivalents thereof limit the scope of a claim to the specifiedmaterials or steps and those that do not materially affect the basic andnovel characteristic or characteristics of the claimed disclosure.

As used herein, the term “proximal” shall refer broadly to the conceptof a portion nearest an origin.

As used herein, the term “distal” shall generally refer to the oppositeof proximal, and thus to the concept of a portion farther from anorigin, or a furthest portion, depending upon the context.

As used herein, color sensors or multi spectrum sensors are thosesensors known to have a color filter array (CFA) thereon so as to filterthe incoming electromagnetic radiation into its separate components. Inthe visual range of the electromagnetic spectrum, such a CFA may bebuilt on a Bayer pattern or modification thereon in order to separategreen, red and blue spectrum components of the light.

Modern digital video systems such as those employed for endoscopyincorporate various levels of calibration for the purpose of renderingthe image as ideal as possible. In essence, the prime motivation is tomimic the human visual system as closely as possible. Raw color imagescaptured under different types of broad-spectrum illumination (such assunlight, tungsten filaments, fluorescent lighting, white LEDs etc.),will all have different overall color casts. The human visual system ishighly effective in automatically balancing out the biases introduced bythe illumination spectra, so as to, e.g., idealize the perception ofwhite and grey scene components. For example, a white sheet of paperalways seems white, irrespective of whether the light is, e.g.,incandescent or daylight. Raw digital images of a white sheet of papermay appear different shades of off-white under different illuminants,however. To counter this, a digital imaging system, as opposed to thehuman visual system, must incorporate a white balance process. In fact,most of the process of white balancing is to adjust for the fact that animage sensor response for each color channel is different. The quantumefficiency for a silicon photodiode or other light sensing element islower for blue photons than for red and green photons for instance.

Endoscopy systems, such as those illustrated in FIG. 1, may have anadvantage in that the illuminant does not change during operation,necessitating only a single calibration step. For continuous, broad bandillumination-based systems, this is may be accomplished by directing theendoscope at a flat, white target, acquiring one or more images,computing the relative red, green and blue signals and storing theresult in memory. During operation, digital gain factors may be appliedto the three color channels in the image signal processor (ISP) tocompensate for the observed relative responses. If the illumination isprovided by pulsing three different wavelengths of monochromatic light(e.g., red, green, and blue), there are two other options for theapplication of white balance:

-   -   Option 1—The light pulse energies are modulated to provide equal        response for the red, green, and blue components; and    -   Option 2—The light energies are maximized to take full benefit        of the dynamic range of the system, without saturating it. Then        the components that have been exaggerated in the optical domain        have appropriate digital attenuation factors applied in the ISP.

Option 2, has an advantage for signal-to-noise ratio, since the dominantsource of noise is the Poisson uncertainty in photon arrival rate, whichscales as the square root of the signal.

The digital processing stages associated with CMOS image sensors arealso concerned with correcting for non-idealities that are inherentwithin the sense technology. One such non-ideality is so-called fixedpattern noise (FPN), which has a strongly detrimental effect on imagequality. It arises due to random variations in black level from pixel topixel. There may also be a column to column component (CFPN) reflectingthe analog readout architecture. The degree to which the FPN isoffensive to an image signal is dependent on the level of contrast withrespect to the true noise sources, such as temporal read noise andphoton shot noise. The perception threshold for random pixel FPN isaround ¼ of the temporal noise at 60 frames per second, while for CFPNit is around 1/20.

In striving for these targets, a strategy may include compensating forFPN using a dark reference buffer stored on the camera or imagingdevice, which may be accessible by the ISP. As each physical pixel issampled by the ISP it may have its dedicated black correction applied.If the illumination is under the fast control of the camera (madepossible with LEDs and laser diodes), periodic dark frames may beacquired to keep a running average of the black offsets in order toaccount for temperature variations. An important component of FPN arisesfrom thermal carrier generation with the photosensitive elements, whichhas an exponential dependence on absolute temperature.

This disclosure is concerned with a convenient method for thecalibration, both initially and at other times during a surgicalprocedure, for endoscopy systems having full control over theirillumination sources. Although an example supported in this disclosureis with a single-use system with sensor at the distal tip, thistechnique is applicable to re-posable, re-usable and limited useendoscopes, with sensor at the distal tip or within the proximal camerahead, with multiple sensor (e.g., for 3D imaging) or single sensor, withrigid or flexible scopes. A set of various system configurations forMinimally Invasive Surgery (MIS) and endoscopy is shown in FIG. 1.

As illustrated in FIG. 1, it will be appreciated that a system 100 fordigital imaging for use in ambient light deficient environments maycomprise a controlled source of electromagnetic radiation 114, an imagesensor 108 comprising a pixel array, which senses reflectedelectromagnetic radiation, optics for continuously focusing a scene ontosaid pixel array (e.g., optics located distally of the image sensor 108in the tip of the endoscope), an endoscope comprising a lumen 102 forallowing electromagnetic energies to reach the pixel array, a cap 230for covering a distal end of the lumen to prevent electromagneticenergies from entering the lumen (see FIGS. 2A and 2B).

It will be appreciated that a dark frame may be created from a singlesensing of the pixel array while the cap 230 is covering the distal endof the lumen. It will be appreciated that the cap 230 may be configured,dimensioned, sized and shaped to fit snuggly onto the distal end of thelumen (illustrated best in FIGS. 2A and 2B). The cap may be made of acompliant material. The cap may be opaque to the electromagneticradiation emitted by an emitter.

The endoscope may be a reusable endoscopic device, a limited useendoscopic device, a re-posable use endoscopic device, or a single-useendoscopic device without departing from the scope of the disclosure.

Continuing to refer to FIG. 1, the system 100 may comprise manydifferent configurations. One example is shown in 1A of FIG. 1, whichillustrates an endoscopic system 100 comprising a rigid angled scope102, an optical coupler 104, a handpiece 106, an image sensor 108, whichmay be located within the handpiece 106 or distally at a tip of theendoscope 102 as illustrated in dashed lines, an electronic cable 110, alight cable 112, such as a fiber optic cable, a light source 114, acontrol unit 116, such as a camera control unit (CCU), a video cable 118and a display 120.

The system configuration shown in 1B of FIG. 1 illustrates an endoscopicsystem 100 comprising a rigid angled scope 102, an optical coupler 104,a handpiece 106, an image sensor 108, which may be located within thehandpiece 106 or distally at a tip of the endoscope 102 as illustratedin dashed lines, an electronic cable 110, a light cable 112, such as afiber optic cable, a control unit 116, such as a camera control unit(CCU), with an integrated light source 114, a video cable 118, and adisplay 120.

The system configuration shown in 1C of FIG. 1 illustrates an endoscopicsystem 100 comprising an articulating scope 102, an optical coupler 104,a handpiece 106, an image sensor 108, which may be located within thehandpiece 106 or distally at a tip of the endoscope 102 as illustratedin dashed lines, an electronic cable 110, a light cable 112, such as afiber optic cable, a light source 114, a control unit 116, such as acamera control unit (CCU), a video cable 118 and a display 120.

The system configuration shown in 1D of FIG. 1 illustrates an endoscopicsystem 100 comprising a handpiece 106 with an integrated rigid 0 degreescope 102, an image sensor 108, which may be located within thehandpiece 106 or distally at a tip of the scope 102 as illustrated indashed lines, a combined electronic and light cable 110, a control unit116, such as a camera control unit (CCU) with an integrated light source114, a video cable 118 and a display 120.

The system configuration shown in 1E of FIG. 1 illustrates an endoscopicsystem 100 comprising a handpiece 106 with an integrated rigid angledscope 102 and rotation post 105, an image sensor 108, which may belocated within the handpiece 106 or distally at a tip of the scope 102as illustrated in dashed lines, a combined electronic and light cable110, a control unit 116, such as a camera control unit (CCU) with anintegrated light source 114, a video cable 118 and a display 120.

The system configuration shown in 1F of FIG. 1 illustrates an endoscopicsystem 100 comprising a handpiece 106 with an integrated articulatingscope 102, an image sensor 108, which may be located within thehandpiece 106 or distally at a tip of the scope 102 as illustrated indashed lines, a combined electronic and light cable 110, a control unit116, such as a camera control unit (CCU) with an integrated light source114, a video cable 118 and a display 120.

The system configuration shown in 1G of FIG. 1 illustrates an endoscopicsystem 100 comprising a handpiece 106 with an integrated flexible scope102, an image sensor 108, which may be located within the handpiece 106or distally at a tip of the scope 102 as illustrated in dashed lines, acombined electronic and light cable 110, a control unit 116, such as acamera control unit (CCU) with an integrated light source 114, a videocable 118 and a display 120.

It will be appreciated that any of the above-identified configurationsfor an endoscopic system shown in FIG. 1, any combination of the aboveelements in a different configuration, and any other configuration usedfor Minimally Invasive Surgery, fall within the scope of thisdisclosure.

Referring now to FIGS. 2A and 2B, in an embodiment, a single-use system,which may come sterile to the Operating Room (OR), might be equippedwith a white cap or an opaque white cap 230 that covers a distal tip 204of an endoscope 202. Upon opening of the sterile package, the endoscopeassembly may be connected to a camera control unit, and the cameracontrol unit may initiate a calibration procedure. During thecalibration procedure the system will reach its quiescent and operatingtemperature and therefore can be optimally calibrated before use. Thecap 230 may simply be removed before starting the procedure.

In an embodiment, a manual procedure, where the operator may place aspecially designed cap 230 over the endoscope distal tip at any timeduring a procedure and then instruct the system to perform thecalibration.

It will be appreciated that the camera system may acquire a number offrames in darkness, e.g., dark frames or dark frame references, to formthe seed dark correction data used for FPN cancellation. The system mayturn on the light source, which feeds light out through the endoscopictip, as it is during normal imaging operation, and acquire another setof frames for the purpose of computing the relative color channelresponses. The system may record these responses in memory, retrieve theresponses from memory, and use them to compute the appropriatecoefficients for white balance. The operator may remove the cap 230 andbegin using the system as normal.

FIGS. 2A and 2B show the distal end 204 of the endoscope 202 covered bythe cap 230 during the dark calibration and light calibration,respectively. FIG. 2B illustrates the light cone 250 illuminating orshowing through the distal end 204 of the endoscope 202 to the cap 230.

It will be appreciated that implementations of the disclosure maycomprise or utilize a special purpose or general-purpose computerincluding computer hardware, such as, for example, one or moreprocessors and system memory, as discussed in greater detail below.Implementations within the scope of the disclosure may also includephysical and other computer-readable media for carrying or storingcomputer-executable instructions and/or data structures. Suchcomputer-readable media can be any available media that can be accessedby a general purpose or special purpose computer system.Computer-readable media that store computer-executable instructions arecomputer storage media (devices). Computer-readable media that carrycomputer-executable instructions are transmission media. Thus, by way ofexample, and not limitation, implementations of the disclosure cancomprise at least two distinctly different kinds of computer-readablemedia: computer storage media (devices) and transmission media.

Computer storage media (devices) includes RAM, ROM, EEPROM, CD-ROM,solid state drives (“SSDs”) (e.g., based on RAM), Flash memory,phase-change memory (“PCM”), other types of memory, other optical diskstorage, magnetic disk storage or other magnetic storage devices, or anyother medium which can be used to store desired program code means inthe form of computer-executable instructions or data structures andwhich can be accessed by a general purpose or special purpose computer.

A “network” is defined as one or more data links that enable thetransport of electronic data between computer systems and/or modulesand/or other electronic devices. In an implementation, a sensor andcamera control unit may be networked in order to communicate with eachother, and other components, connected over the network to which theyare connected. When information is transferred or provided over anetwork or another communications connection (either hardwired,wireless, or a combination of hardwired or wireless) to a computer, thecomputer properly views the connection as a transmission medium.Transmissions media can include a network and/or data links which can beused to carry desired program code means in the form ofcomputer-executable instructions or data structures and which can beaccessed by a general purpose or special purpose computer. Combinationsof the above should also be included within the scope ofcomputer-readable media.

As can be seen in FIG. 3, various computer system components, programcode means in the form of computer-executable instructions or datastructures that can be transferred automatically from transmission mediato computer storage media (devices) (or vice versa). For example,computer-executable instructions or data structures received over anetwork or data link can be buffered in RAM within a network interfacemodule (e.g., a “NIC”), and then eventually transferred to computersystem RAM and/or to less volatile computer storage media (devices) at acomputer system. RAM can also include solid state drives (SSDs or PCIxbased real time memory tiered Storage, such as FusionIO). Thus, itshould be understood that computer storage media (devices) can beincluded in computer system components that also (or even primarily)utilize transmission media.

Computer-executable instructions comprise, for example, instructions anddata which, when executed at a processor, cause a general purposecomputer, special purpose computer, or special purpose processing deviceto perform a certain function or group of functions. The computerexecutable instructions may be, for example, binaries, intermediateformat instructions such as assembly language, or even source code.Although the subject matter has been described in language specific tostructural features and/or methodological acts, it is to be understoodthat the subject matter defined in the appended claims is notnecessarily limited to the described features or acts described above.Rather, the described features and acts are disclosed as example formsof implementing the claims.

Those skilled in the art will appreciate that the disclosure may bepracticed in network computing environments with many types of computersystem configurations, including, personal computers, desktop computers,laptop computers, message processors, control units, camera controlunits, hand-held devices, hand pieces, multi-processor systems,microprocessor-based or programmable consumer electronics, network PCs,minicomputers, mainframe computers, mobile telephones, PDAs, tablets,pagers, routers, switches, various storage devices, and the like. Itshould be noted that any of the above mentioned computing devices may beprovided by or located within a brick and mortar location. Thedisclosure may also be practiced in distributed system environmentswhere local and remote computer systems, which are linked (either byhardwired data links, wireless data links, or by a combination ofhardwired and wireless data links) through a network, both performtasks. In a distributed system environment, program modules may belocated in both local and remote memory storage devices.

Further, where appropriate, functions described herein can be performedin one or more of: hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) or field programmable gate arrays can beprogrammed to carry out one or more of the systems and proceduresdescribed herein. Certain terms are used throughout the followingdescription and Claims to refer to particular system components. As oneskilled in the art will appreciate, components may be referred to bydifferent names. This document does not intend to distinguish betweencomponents that differ in name, but not function.

FIG. 3 is a block diagram illustrating an example computing device 300.Computing device 300 may be used to perform various procedures, such asthose discussed herein. Computing device 300 can function as a server, aclient, or any other computing entity. Computing device can performvarious monitoring functions as discussed herein, and can execute one ormore application programs, such as the application programs describedherein. Computing device 300 can be any of a wide variety of computingdevices, such as a desktop computer, a notebook computer, a servercomputer, a handheld computer, camera control unit, tablet computer andthe like.

Computing device 300 includes one or more processor(s) 302, one or morememory device(s) 304, one or more interface(s) 306, one or more massstorage device(s) 308, one or more Input/Output (I/O) device(s) 310, anda display device 330 all of which are coupled to a bus 312. Processor(s)302 include one or more processors or controllers that executeinstructions stored in memory device(s) 304 and/or mass storagedevice(s) 308. Processor(s) 302 may also include various types ofcomputer-readable media, such as cache memory.

Memory device(s) 304 include various computer-readable media, such asvolatile memory (e.g., random access memory (RAM) 314) and/ornonvolatile memory (e.g., read-only memory (ROM) 316). Memory device(s)304 may also include rewritable ROM, such as Flash memory.

Mass storage device(s) 308 include various computer readable media, suchas magnetic tapes, magnetic disks, optical disks, solid-state memory(e.g., Flash memory), and so forth. As shown in FIG. 3, a particularmass storage device is a hard disk drive 324. Various drives may also beincluded in mass storage device(s) 308 to enable reading from and/orwriting to the various computer readable media. Mass storage device(s)308 include removable media 326 and/or non-removable media.

I/O device(s) 310 include various devices that allow data and/or otherinformation to be input to or retrieved from computing device 300.Example I/O device(s) 310 include digital imaging devices,electromagnetic sensors and emitters, cursor control devices, keyboards,keypads, microphones, monitors or other display devices, speakers,printers, network interface cards, modems, lenses, CCDs or other imagecapture devices, and the like.

Display device 330 includes any type of device capable of displayinginformation to one or more users of computing device 300. Examples ofdisplay device 330 include a monitor, display terminal, video projectiondevice, and the like.

Interface(s) 306 include various interfaces that allow computing device300 to interact with other systems, devices, or computing environments.Example interface(s) 306 may include any number of different networkinterfaces 320, such as interfaces to local area networks (LANs), widearea networks (WANs), wireless networks, and the Internet. Otherinterface(s) include user interface 318 and peripheral device interface322. The interface(s) 306 may also include one or more user interfaceelements 318. The interface(s) 306 may also include one or moreperipheral interfaces such as interfaces for printers, pointing devices(mice, track pad, etc.), keyboards, and the like.

Bus 312 allows processor(s) 302, memory device(s) 304, interface(s) 306,mass storage device(s) 308, and I/O device(s) 310 to communicate withone another, as well as other devices or components coupled to bus 332.Bus 312 represents one or more of several types of bus structures, suchas a system bus, PCI bus, IEEE 1394 bus, USB bus, and so forth.

For purposes of illustration, programs and other executable programcomponents are shown herein as discrete blocks, although it isunderstood that such programs and components may reside at various timesin different storage components of computing device 300, and areexecuted by processor(s) 302. Alternatively, the systems and proceduresdescribed herein can be implemented in hardware, or a combination ofhardware, software, and/or firmware. For example, one or moreapplication specific integrated circuits (ASICs) can be programmed tocarry out one or more of the systems and procedures described herein.

It will be appreciated that the disclosure may be used with any imagesensor, whether a CMOS image sensor or CCD image sensor, withoutdeparting from the scope of the disclosure. Further, the image sensormay be located in any location within the overall system, including, butnot limited to, the tip of the endoscope, the hand piece of the imagingdevice or camera, the control unit, or any other location within thesystem without departing from the scope of the disclosure.

Implementations of an image sensor that may be utilized by thedisclosure include, but are not limited to, the following, which aremerely examples of various types of sensors that may be utilized by thedisclosure.

Referring now to FIGS. 4A and 4B, the figures illustrate a perspectiveview and a side view, respectively, of an implementation of a monolithicsensor 400 having a plurality of pixel arrays for producing a threedimensional image in accordance with the teachings and principles of thedisclosure. Such an implementation may be desirable for threedimensional image capture, wherein the two pixel arrays 402 and 404 maybe offset during use. In another implementation, a first pixel array 402and a second pixel array 404 may be dedicated to receiving apredetermined range of wave lengths of electromagnetic radiation,wherein the first pixel array 402 is dedicated to a different range ofwave length electromagnetic radiation than the second pixel array 404.

FIGS. 5A and 5B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor 500 built on aplurality of substrates. As illustrated, a plurality of pixel columns504 forming the pixel array are located on the first substrate 502 and aplurality of circuit columns 508 are located on a second substrate 506.Also illustrated in the figure are the electrical connection andcommunication between one column of pixels to its associated orcorresponding column of circuitry. In one implementation, an imagesensor, which might otherwise be manufactured with its pixel array andsupporting circuitry on a single, monolithic substrate/chip, may havethe pixel array separated from all or a majority of the supportingcircuitry. The disclosure may use at least two substrates/chips, whichwill be stacked together using three-dimensional stacking technology.The first 502 of the two substrates/chips may be processed using animage CMOS process. The first substrate/chip 502 may be comprised eitherof a pixel array exclusively or a pixel array surrounded by limitedcircuitry. The second or subsequent substrate/chip 506 may be processedusing any process, and does not have to be from an image CMOS process.The second substrate/chip 506 may be, but is not limited to, a highlydense digital process in order to integrate a variety and number offunctions in a very limited space or area on the substrate/chip, or amixed-mode or analog process in order to integrate for example preciseanalog functions, or a RF process in order to implement wirelesscapability, or MEMS (Micro-Electro-Mechanical Systems) in order tointegrate MEMS devices. The image CMOS substrate/chip 502 may be stackedwith the second or subsequent substrate/chip 506 using anythree-dimensional technique. The second substrate/chip 506 may supportmost, or a majority, of the circuitry that would have otherwise beenimplemented in the first image CMOS chip 502 (if implemented on amonolithic substrate/chip) as peripheral circuits and therefore haveincreased the overall system area while keeping the pixel array sizeconstant and optimized to the fullest extent possible. The electricalconnection between the two substrates/chips may be done throughinterconnects 503 and 505, which may be wirebonds, bump and/or TSV(Through Silicon Via).

FIGS. 6A and 6B illustrate a perspective view and a side view,respectively, of an implementation of an imaging sensor 600 having aplurality of pixel arrays for producing a three dimensional image. Thethree dimensional image sensor may be built on a plurality of substratesand may comprise the plurality of pixel arrays and other associatedcircuitry, wherein a plurality of pixel columns 604 a forming the firstpixel array and a plurality of pixel columns 604 b forming a secondpixel array are located on respective substrates 602 a and 602 b,respectively, and a plurality of circuit columns 608 a and 608 b arelocated on a separate substrate 606. Also illustrated are the electricalconnections and communications between columns of pixels to associatedor corresponding column of circuitry.

FIGS. 7-9 are flow charts illustrating implementations of a system andmethod for adjusting white balance in accordance with the principles andteachings of the disclosure.

The system and method 700 of FIG. 7 for adjusting fixed pattern noise ina digital imaging process for use with an endoscope in ambient lightdeficient environments may comprise controlling emissions ofelectromagnetic radiation using a controlled source of electromagneticradiation at 710. At 720, the system and method 700 may continuouslyfocus a scene onto a pixel array. At 730, the system and method 700 maycomprise sensing reflected electromagnetic radiation with the pixelarray to create an image frame. At 740, the system and method 700 maycomprise placing a cap over a lumen such that the cap is blockingexternal illumination to each pixel in the pixel array. At 750, thesystem and method 700 may comprise sensing the pixel array while the capis in place with no applied electromagnetic radiation. At 760, thesystem and method 700 may comprise creating a dark frame reference foruse in removing fixed pattern noise. At 770, the system and method 700may comprise comparing the dark frame reference to the image frame. At780, the system and method 700 may comprise correcting the image frameby removing fixed pattern noise from an image using the dark framereference. At 790, the system and method 700 may comprise creating astream of images by combining a plurality of image frames to form avideo stream.

The system and method 700 may comprise actuating an emitter to emit apulse of a wavelength of electromagnetic radiation to cause illuminationwithin the light deficient environment. In an implementation, the pulsemay be within a wavelength range that comprises a portion of theelectromagnetic spectrum. In an implementation, the emitter may be alaser emitter and the system and method 700 may further comprise pulsingthe laser emitter at a predetermined interval. In an implementation, themethod may further comprise actuating a pixel array at a sensinginterval that corresponds to the pulse interval of the laser emitter.

The system and method 700 may comprise creating a dark frame from asingle sensing of the pixel array while the cap is in place. It will beappreciated that in an implementation, a plurality of dark frames may becreated from a plurality of sensing the pixel array while the cap is inplace. In an implementation, the dark frames may be created upon startupof a system and stored within memory associated with the system.

In an implementation, the system and method 700 may comprise creating adark frame after the pixel array is at operating temperature. In animplementation, the dark frame may be created after a surgical procedurehas begun. In an implementation, the dark frame may comprise a pluralityof dark frames that are created as part of the image frame stream by notapplying electromagnetic radiation at given times and stored withinmemory associated with the system.

The system and method 700 may comprise a plurality of caps thatcorrespond and are opaque to the emitted electromagnetic radiation.

The system and method 700 may comprise a response of the pixel arraythat corresponds to a photo-signal generated under controlledmonochromatic radiation. In an implementation, the system and method 700may comprise a response of the pixel array corresponds to thephoto-signal generated under a plurality of wavelengths of radiation. Inan implementation, a response of the pixel array corresponds to thephoto-signal generated under a continuous spectrum of wavelengths ofradiation.

The system and method 800 of FIG. 8 for adjusting fixed pattern noise ina digital imaging process for use with an endoscope in ambient lightdeficient environments may comprise actuating a laser emitter to emit apulse of a wavelength of electromagnetic radiation to cause illuminationwithin the light deficient environment at 810. At 820, the system andmethod 800 may pulse the laser emitter at a predetermined interval. At830, the system and method 800 may comprise sensing reflectedelectromagnetic radiation from the pulse with a pixel array to create animage frame. At 840, the system and method 800 may comprise actuatingthe pixel array at a sensing interval that corresponds to the pulseinterval of the laser emitter. At 850, the system and method 800 maycomprise placing a cap over a lumen that is allowing or permitting theelectromagnetic energies to each pixel in the pixel array. At 860, thesystem and method 800 may comprise sensing the pixel array while the capis in place. In an implementation, sensing the pixel array while the capis in place includes sensing with no applied electromagnetic radiation.At 870, the system and method 800 may comprise creating a dark framereference for use in removing fixed pattern noise. At 880, the systemand method 800 may comprise correcting the image frame by removing noisecorresponding to the dark frame reference. At 890, the system and method800 may comprise creating a stream of images by combining a plurality ofimage frames to form a video stream.

The system and method 900 of FIG. 9 for adjusting white balance in adigital imaging process for use with an endoscope in ambient lightdeficient environments may comprise illuminating a scene using acontrolled source of electromagnetic radiation at 910. At 920, thesystem and method 900 may comprise continuously focusing the scene ontoa pixel array. At 930, the system and method 900 may comprise sensingreflected electromagnetic radiation sensed by the pixel array. At 940,the system and method 900 may comprise placing a cap over a lumen, suchthat the cap is blocking external illumination to each pixel in thepixel array. At 950, the system and method 900 may comprise sensing thepixel array while the cap is in place with applied electromagneticradiation. At 960, the system and method 900 may comprise measuring aresponse from the pixel array with applied electromagnetic radiation andoutputting a result in memory. At 970, the system and method 900 maycomprise using the result to compute white balance coefficients. At 980,the system and method 900 may comprise white balancing the system usingthe white balance coefficients. At 990, the system and method 900 maycomprise creating a stream of images by combining a plurality of imageframes to form a video stream.

In an implementation, the system and method 900 may comprise a pluralityof caps that correspond and are opaque to the emitted electromagneticradiation.

In an implementation, a response of the pixel array may correspond to aphoto-signal generated under controlled monochromatic radiation. In animplementation, a response of the pixel array corresponds to thephoto-signal generated under a plurality of wavelengths of radiation. Inan implementation, a response of the pixel array corresponds to thephoto-signal generated under a continuous spectrum of wavelengths ofradiation.

In an implementation, the endoscope may be a reusable endoscopic device.In an implementation, the endoscope is a limited use endoscopic device.In an implementation, the endoscope is a re-posable use endoscopicdevice. In an implementation, the endoscope is a single-use endoscopicdevice.

In an implementation, the system and method 900 may comprise actuating alaser emitter to emit a pulse of a wavelength of electromagneticradiation to cause illumination within the light deficient environment.In an implementation, the pulse is within a wavelength range thatcomprises a portion of the electromagnetic spectrum. In animplementation, the system and method 900 may further comprise pulsingthe laser emitter at a predetermined interval. In an implementation, thesystem and method 900 may comprise actuating a pixel array at a sensinginterval that corresponds to the pulse interval of said laser emitter.

It will be appreciated that the teachings and principles of thedisclosure may be used in a reusable device platform, a limited usedevice platform, a re-posable use device platform, or asingle-use/disposable device platform without departing from the scopeof the disclosure. It will be appreciated that in a re-usable deviceplatform an end-user is responsible for cleaning and sterilization ofthe device. In a limited use device platform the device can be used forsome specified amount of times before becoming inoperable. Typical newdevice is delivered sterile with additional uses requiring the end-userto clean and sterilize before additional uses. In a re-posable usedevice platform a third-party may reprocess the device (e.g., cleans,packages and sterilizes) a single-use device for additional uses at alower cost than a new unit. In a single-use/disposable device platform adevice is provided sterile to the operating room and used only oncebefore being disposed of.

Additionally, the teachings and principles of the disclosure may includeany and all wavelengths of electromagnetic energy, including the visibleand non-visible spectrums, such as infrared (IR), ultraviolet (UV), andX-ray.

It will be appreciated that various features disclosed herein providesignificant advantages and advancements in the art. The followingembodiments are exemplary of some of those features.

In the foregoing Detailed Description of the Disclosure, variousfeatures of the disclosure are grouped together in a single embodimentfor the purpose of streamlining the disclosure. This method ofdisclosure is not to be interpreted as reflecting an intention that theclaimed disclosure requires more features than are expressly recited ineach claim. Rather, inventive aspects lie in less than all features of asingle foregoing disclosed embodiment.

It is to be understood that the above-described arrangements are onlyillustrative of the application of the principles of the disclosure.Numerous modifications and alternative arrangements may be devised bythose skilled in the art without departing from the spirit and scope ofthe disclosure and the appended claims are intended to cover suchmodifications and arrangements.

Thus, while the disclosure has been shown in the drawings and describedabove with particularity and detail, it will be apparent to those ofordinary skill in the art that numerous modifications, including, butnot limited to, variations in size, materials, shape, form, function andmanner of operation, assembly and use may be made without departing fromthe principles and concepts set forth herein.

Further, where appropriate, functions described herein can be performedin one or more of: hardware, software, firmware, digital components, oranalog components. For example, one or more application specificintegrated circuits (ASICs) can be programmed to carry out one or moreof the systems and procedures described herein. Certain terms are usedthroughout the following description and Claims to refer to particularsystem components. As one skilled in the art will appreciate, componentsmay be referred to by different names. This document does not intend todistinguish between components that differ in name, but not function.

1-37. (canceled)
 38. A method for adjusting fixed pattern noise in adigital imaging process for use with an endoscope in ambient lightdeficient environments comprising: providing, during a training period,a cap that fits over a lumen of the endoscope such that the cap isblocking external illumination to each pixel in a pixel array; creating,during the training period, a first dark frame reference for use inremoving fixed pattern noise while the cap is in place with no appliedelectromagnetic radiation and at a quiescent operating temperature forthe endoscope; removing the cap from the lumen at the quiescentoperating temperature for the endoscope; in response to removing the capfrom the lumen, sensing electromagnetic radiation with said pixel arrayto create an image frame; and correcting the image frame by removingfixed pattern noise from an image using the first dark frame reference.39. The method of claim 38, wherein the method further comprisesactuating an emitter to emit a pulse of a wavelength of electromagneticradiation to cause illumination within the light deficient environment.40. The method of claim 39, wherein said pulse is within a wavelengthrange that comprises a portion of the electromagnetic spectrum.
 41. Themethod of claim 40, wherein said emitter is a laser emitter and whereinthe method further comprises pulsing said laser emitter at apredetermined interval.
 42. The method of claim 41, wherein the methodfurther comprises actuating a pixel array at a sensing interval thatcorresponds to the pulse interval of said laser emitter.
 43. The methodof claim 38, wherein the first dark frame is created from a singlesensing of the pixel array while the cap is in place.
 44. The method ofclaim 38, wherein a plurality of dark frames are created from aplurality of sensing the pixel array while the cap is in place.
 45. Themethod of claim 44, wherein said dark frames are created upon startup ofa system and stored within memory associated with the system.
 46. Themethod of claim 38, wherein the first dark frame is created before thesurgical procedure has begun, the method further comprising; in responseto removing the cap from the lumen, beginning the surgical procedureusing the endoscope; placing the cap on the lumen after beginning thesurgical procedure; in response to placing the cap on the lumen,creating a second dark frame reference for use in removing fixed patternnoise while the cap is in place after the surgical procedure has begunwith no applied electromagnetic radiation; and correcting the imageframe by removing fixed pattern noise from an image using the seconddark frame reference.
 47. The method of claim 38, wherein the first darkframe comprises a plurality of dark frames that are created as part ofthe image frame stream by not applying electromagnetic radiation atgiven times and stored within memory associated with the system.
 48. Themethod of claim 38, wherein sensing electromagnetic radiation comprisessensing controlled monochromatic radiation to generate the image frame.49. The method of claim 38, wherein sensing electromagnetic radiationcomprises sensing under a plurality of wavelengths of radiation togenerate the image frame.
 50. The method of claim 38, wherein sensingelectromagnetic radiation comprises sensing under a continuous spectrumof wavelengths of radiation to generate the image frame.
 51. A systemfor digital imaging for use in ambient light deficient environmentscomprising: an endoscope comprising a lumen and an image sensor, whereinthe image sensor comprises a pixel array that senses electromagneticradiation; a cap configured to selectively cover a distal end of thelumen to block external illumination to each pixel in said pixel array;a camera control unit in electrical communication with the image sensor;wherein the image sensor creates a dark frame reference, during atraining period, for use in removing fixed pattern noise while the capis in place with no applied electromagnetic radiation at a quiescentoperating temperature, and, in response to removing the cap from thelumen at the quiescent operating temperature, senses electromagneticradiation with said pixel array to create an image frame; and whereinthe camera control unit corrects the image frame by removing fixedpattern noise from an image using the dark frame reference.
 52. Thesystem of claim 51, wherein a dark frame is created from a singlesensing of the pixel array while the cap is covering the distal end. 53.The system of claim 51, wherein the cap is sized and shaped to fitsnuggly onto the lumen.
 54. The system of claim 51, wherein the cap ismade of a compliant material.
 55. The system of claim 51, wherein thecap is opaque to the electromagnetic radiation emitted by an emitter.56. The system of claim 51, wherein the endoscope is a reusableendoscopic device.
 57. The system of claim 51, wherein the endoscope isa limited use endoscopic device.
 58. The system of claim 51, wherein theendoscope is a re-posable use endoscopic device.
 59. The system of claim51, wherein the endoscope is a single-use endoscopic device.
 60. Amethod for adjusting fixed pattern noise in a digital imaging processfor use with an endoscope in ambient light deficient environmentscomprising: providing, during a training period, a cap that fits over alumen of the endoscope such that the cap is blocking externalillumination to each pixel in a pixel array; creating a dark framereference, during a training period, for use in removing fixed patternnoise while the cap is in place with no applied electromagneticradiation at a quiescent operating temperature; illuminating, during thetraining period, an interior of the cap using a controlled source ofelectromagnetic radiation; determining, during the training period,white balance coefficients while the cap is in place and duringillumination of the interior by a source of electromagnetic illuminationat a quiescent operating temperature; removing the cap from the lumen atthe quiescent operating temperature for the endoscope; in response toremoving the cap from the lumen, sensing electromagnetic radiation withsaid pixel array to create an image frame; and correcting the imageframe using the dark frame reference and the white balance coefficients.61. The method of claim 60, wherein sensing electromagnetic radiationcomprises sensing controlled monochromatic radiation to generate theimage frame.
 62. The method of claim 60, wherein sensing electromagneticradiation comprises sensing under a plurality of wavelengths ofradiation to generate the image frame.
 63. The method of claim 60,wherein sensing electromagnetic radiation comprises sensing under acontinuous spectrum of wavelengths of radiation to generate the imageframe.
 64. The method of claim 60, wherein the endoscope is a reusableendoscopic device.
 65. The method of claim 60, wherein the endoscope isa limited use endoscopic device.
 66. The method of claim 60, wherein theendoscope is a re-posable use endoscopic device.
 67. The method of claim60, wherein the endoscope is a single-use endoscopic device.
 68. Themethod of claim 60, wherein the controlled source of electromagneticradiation comprises a laser emitter, the method further comprisingemitting a pulse of a wavelength of electromagnetic radiation to causeillumination within the light deficient environment, wherein sensingelectromagnetic radiation to create the image frame comprises sensingduring the emitting the pulse.
 69. The method of claim 68, wherein themethod further comprises pulsing said laser emitter at a predeterminedinterval.
 70. The method of claim 69, wherein the method furthercomprises sensing electromagnetic radiation using the pixel array at asensing interval that corresponds to the pulse interval of the laseremitter.